Deep Visual Constraints: Neural Implicit Models for Manipulation Planning From Visual Input

Manipulation planning is the problem of finding a sequence robot configurations that involves interactions with objects in scene, e.g., grasping and placing an object, or more general tool-use. To achieve such interactions, traditional approaches require hand-engineering object representations interaction constraints, which easily becomes tedious when complex objects/interactions are considered. Inspired by recent advances 3D modeling, e.g. NeRF, we propose method to represent as continuous functions upon constraint features defined jointly trained. In particular, proposed pixel-aligned representation directly inferred from images known camera geometry naturally acts perception component whole manipulation pipeline, thereby enabling long-horizon only visual input .